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I, LILY KNEZEVICH, hereby declare the following: 

1 . I am an employee of the Law School Admission Council ("LS AC"), assignee of the 
present application. 

2. I am familiar with the publication entitled Armstrong, R., Weissman A. and Belov, D., 
Developing and Assembling the Law School Admission Test ("Developing and Assembling the 



3. Developing and Assembling the LSAT refers to a 2002 implementation of an LSAT 
assembler utilizing an adaptive stochastic search approach. The 2002 implementation referred to 
in this article was sometimes referred to internally by LSAC as the "automated assembler." 

4. In order for a test assembly algorithm to be used in connection with assembling an actual 
LSAT to be taken by students, it is necessary to build an operational interface between the test 
assembler and what LSAC refers to as the ITEMS architecture, which is essentially a computer 
system that maintains the actual test questions (i.e., items) that may appear on the LSAT. 

5. Attached hereto as Exhibit A is a document printed on February 16, 2010, from LSAC's 
computer system. This document shows the tasks that are to be completed in connection with 
building the interface between the test assembler and the ITEMS architecture. As can be seen, 
the exemplary time line shown in Exhibit A begins in September 2002 and ends over a year later. 
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6. Exhibit A notes that between September 16, 2002 and January 1 5, 2003, the Test 
Assembly algorithm (the predecessor to the automated assembler) was to be evaluated. Indeed, 
during this time period, at my retjuest, Dmitry Belov (the inventor of the present application) 
evaluated the Test Assembly algorithm, identified certain weaknesses in it, and developed the 
automated assembler to be used in its stead. 

7. Once the automated assembler had been developed, the various tasks required to build th 
interface between the automated assembler and the ITEMS architecture (as identified in Exhibit 
A) were carried out 

8. Exhibit B is a Memorandum dated September 26, 2003, on which I was copied. Exhibit 
B describes an analysis of LSAT forms 69, 70, 71, and 72 (i.e., each LSAT form that is to be 
taken by students is assigned a unique form number). The analysis for Form 69 refers to the 
automated assembler. 

9. Form 69 was the first LSAT form, that was to be taken by students, to be assembled 
using the automated assembler, as evidenced by Exhibit C, Exhibit C is a memorandum dated 
September 18, 2002, on which I was copied. Exhibit C describes an analysis of LSAT forms 65, 
66, 67 and 68. The analysis of each of these forms refers to the TestBuilder assembly (i.e., the 
predecessor to the automated assembler). 

10. Exhibit B demonstrates that, at least as of September 26, 2003, LSAT Form 69 was still 
being analyzed and had not yet been taken by students. 

I hereby declare that all statements made herein of my own knowledge are true and that 
all statements made on information and belief are believed to be true; and further that these 
statements are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code, and that such willful statements may jeopardize the validity of the 
application or any patent issued thereon. 
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EXHIBIT A 



EXHIBIT B 



Memorandum 



To: J, Lorifi, S. Luebke, G. Plumer 

CC: R. Adams, L. Knezevich, form files 

From: Dave Kary 

Date: 09/26/03 

Re: Preliminary Assemblies of forms 69,70, 71, and 72. 



Form 69: 

100 items, predicted mean score 
AR: 22 items 
LA: 25 items 
distribution requirement.) 
LB: 25 items. This section has 
RC: 28 items. One item from the initial assembly 
because it was part of an enemy pair in this section. 



(The automated assembler does not enforce a key 



^was replace* 



Form 70: 

100 items, predicted mean score 
AR: 22 items. 



One of the sets 

count maximum, but the total maximum for the section is within spec. 
LA: 25 items 



LB: 25 items. 



violates the word 




RC: 28 items 



Form 71: 

100 items, predicted mean score 
AR: 22 items 
LA: 25 items 
LB: 25 items. 




RC: 28 items. Contains a W set. 



Form 72: 

100 items, predicted mean score 
AR: 22 items 
LA: 25 items, 
LB: 25 items, 

RC: 28 items. Contains a W set 




An important note regarding predicted means: 
These 100 item fonrn were assembled so that their predicted 
"rand ~ 



ineans fall within a range between 




* to m ind ividual section's predicted mean. 



GForm: 

For the purposes of the 
mean score ft 




is interchangeable wii 
The predicted mean score fo 




The predicted 
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AR sets that come from high-difficulty pretests 



This assembly includes three AR sets that 



were pretested in the third or fourth position in pretests that 



proved to be quite difficult. Those sets are listed here along with the IDs of the sections in which they 
were pretested. The pretest sections can be viewed in Item Bank Monitoring, in the Public Subpools 



under DAK. 
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p ■ ■ 


■ 









Ordering within sections: 

Form sections are ordered in accordance with the 11/99 guidelines, "Creating (Initial) Pre-Op Section 
Ordering Summary Sheets." 

Pool usage issues: 

All . items in these four initial assemblies came from an assembly pool of items pretested on or 
before 12/99. The automated assembler was able to produce five forms from this pool; that is, the 
four forms that were chosen for review plus one other. Newly developed pool evaluation tools 
will be run in the near future to determine if any more forms can be drawn from this pool. 



Other issues: 



Fseems to be missing italics. 
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EXHIBIT C 



Memorandum 



To: I Lori6, S. Luebke, G. Plumer 

CC: R - Adams, L. Knezevich, form files 

From: Dave Kary 

Date: 09/18/02 

Re: Preliminary Assemblies of forms 65, 66, 67, and 68 



Form 65: 

AR: 23 items (pretest position for sets: 1,3,3,4), One set from the TestBuilder assembly was 
replaced because it resulted in too manyfp^items. The replacement set 

LA: 25 items 
LB: 26 items 

RC: 27 items, 1 M set, no W set (pretest position for sets: 1,2,3,4) 

Predicted mean for the TestBuilder assembly, according to that program's "Theta Statistics" 
calculations: 4MB 

Change in P+E between the TestBuilder assembly and the revised assembly* 



Form 66: 

AR: 22 items, (pretest position for sets: 1,2,3,4) 
LA: 25 items 



September 18, 2002 



LB: 26 items. 




has an error in its text: " 



RC: 28 items, 1 M set, i W set, (pretest position for sets: 1,2,3,4). In the initial RC 

assembly, two se ts JfcM^BMHMMBMMMM^hMiflB^^Mi n l 

m*m*m^&. (Both were Wsets: two W'sets appeared in the same section because 
one was missing the relevant codes.) For this reason, SMNfewas replaced by 



Predicted mean for the TestBuilder assembly, according to that program's 'Theta Statistics" 
calculations: BBSS? 



Change in total P+E between the TestBuilder assembly and the revised assembly: 



Form 67; 



AR: 23 items (pretest position for sets: 1,2,2,4; 




LA: 25 items 



LB; 26 items. One of the items in this section 



RC: 27 items, ^HHHW"P(pretest position for sets: 1,1,2,4). In the TestBuilder 

assembly, two sets dealt with a similar subject matter (linguistics). As a result^BS^ 
was replaced by^SHBK 

Predicted me an for th e TestBuilder assembly, according to that program's "Theta Statistics" 
calculations: (Hi 



Change in total P+E between the TestBuilder assembly and the revised assembly: 
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September 18. 2002 



Form 68: 

Afc22jtems (pretest position for sets: 1,2,3.4). Note: This section has fiv^6 nd oniv 
one^owever at least one of the^HB^s more like an» 

LA: 25 items 
LB: 26 items 

RC: 28 items, 1 set that is both M and W (pretest position for sets: 1,1,3,4) 

Predicted mean for the TestBuilder assembly, according to that program's "Theta Statistics" 
calculations:jMB 



Change in total P +E between the TestBuilder assembly and the revised assembly: 



GForm: 




eg 

Total P+E: F< 



Ordering within sections: 

When the forms were imported into Item Bank Monitoring, the sections were mechanically 
ordered, following the 1 1/99 guidelines, "Creating (Initial) Pre-Op Section Ordering 
Summary Sheets." Most of this ordering had to be revised because of item/set replacement. 
This was done by the initial assembler, also following the above guidelines. 
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